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Image fusion refers to the techniques that integrate complementary information from multiple image 
sensors’ data in a way that makes the new images more suitable for human visual perception. The paper 
focuses on the low color contrast problem of linear fusion algorithms with color transfer method. Firstly, 
the contrast of infrared and visible images is enhanced using local histogram equalization and median 
filter. Then the two enhanced images are fused into the three components of a Lab image in terms of a 
simple linear fusion strategy. To enhance the color contrast between the target and the background, 
the scaling factor is introduced into the transferring equation in the b channel. Experimental results 
based on three different data sets show that the hot and cold targets are all popped out with intense col¬ 
ors while the background details present natural color appearance. Target detection experiments through 
target recognition area, detection rate, target-background discrimination also show that the presented 
method has a better performance than the former methods. 

© 2011 Elsevier B.V. All rights reserved. 


1. Introduction 

The availability of modern night vision systems, like image 
intensifies and thermal cameras, enables operations during night 
and in adverse weather conditions. The two most common night¬ 
time imaging systems display either emitted infrared radiation 
(IR) or dim reflected light. Generally, the visible and IR images 
are complementary, i.e. the IR image has better hot contrast but 
less details than visible image, the visible image has much more 
plenty high-frequency information but worse target contrast espe¬ 
cially under bad luminance condition. Thus, techniques for fusing 
the two images should be employed in order to provide a compact 
representation of the scene with increased interpretation capabil¬ 
ities. In recent years, fusion of visible and IR images gets increasing 
concerns. Fusion methods include gray fusion (such as principal 
component analysis, contrast modulation, pyramid modulation, 
wavelet transform), and color fusion. In principle, color imagery 
has several benefits over monochrome imagery for surveillance, 
reconnaissance, and security applications. The color level which 
human can distinguish is about hundreds times than gray level 
and many experiments show that color fusion may improve fea¬ 
ture contrast, which allows for better scene segmentation and ob¬ 
ject detection [1,2]. So color fusion is becoming a more and more 
important research field and a number of color fusion methods 
have been proposed [3-6]. 

One key topic of the color fusion is color constancy. It has been 
shown that the lack of color constancy with inappropriate color 
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mapping may hinder human performance [13]. In 2003, Toet 
[7,8] originally apply color transfer technology that ever been used 
in color alteration among visible images [9] into multi-band color 
night vision. This method matches the first order statistical proper¬ 
ties (mean and standard deviation) of night vision imagery to those 
of a target daylight color image. As a result, the color appearance of 
the colorized night vision image resembles that of the natural tar¬ 
get image. But color space transformations between RGB and LMS, 
LMS and toc / 3 , logarithm and exponent operation occupy much 
time. And this method does not provide color constancy for dy¬ 
namic imagery. To overcome above shortcomings, Hogervorst 
and Toet [2] introduce a simple and fast method to consistently ap¬ 
ply natural daytime colors to multi-band night vision imagery. 
Their idea is easily implemented using standard color lookup table 
techniques to optimize the match between the false color fused 
image and the reference image. Once the lookup-table has been de¬ 
rived, the color mapping can be deployed in real-time multi-band 
image sequences of similar scenes. But this method requires multi¬ 
band images and a daytime reference image that are in exact cor¬ 
respondence. And their methods process all three channel of the 
color space with the same linear mapping in the color transfer step, 
resulting in low contrast between the target and the background. 
So targets that are clear in the IR image sometimes may become 
invisible and are difficult to detect, that may hinder target detec¬ 
tion tasks [13]. 

While popping-out targets is another key topic of color night vi¬ 
sion, which enhances human visual capabilities in discriminating 
features form their background and is essential to target detection 
tasks. However, to date, little attention has been paid to this prob¬ 
lem. In 2007 [10], Wang proposed a nonlinear transfer method 
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based on local mean value of the IR after nonlinear fusion. This 
method can pop out the hot targets with intense red color while 
the background rendered natural color appearance. Zheng [11] 
and Ma [12] proposed a novel local color night vision algorithm 
based on image segmentation, recognition and local color transfer 
to enhance the color mapping effect. However, these methods are 
even more expensive than Toet’s original color transfer [7,8], since 
they involve time-consuming procedures such as nonlinear diffu¬ 
sion, local recognition, local comparisons and image segmentation. 
In addition, the nonlinear scheme based on Gaussians convolution 
operation sometimes brings blur effects. This may hinder situa¬ 
tional awareness. In 2010, Yin et al. [13] presented one color con¬ 
trast enhancement method for color night vision. They introduce a 
ratio of local to global divergence of the IR image to improve the 
color contrast in the color transfer process after a simple linear fu¬ 
sion. The experiments show that this method can bring an 
improvement in the hit rate for target detection than the global 
statistic method. 

But due to the bad weather or environment, the night vision 
imagery is degraded by the high noise or low contrast, while the 
above works rarely consider this problem. In addition, Yin’s meth¬ 
od can get improvement in the target detection, but cold targets 
are not highlighted effectively and some backgrounds have unnat¬ 
urally colors in the final fused color image, i.e. road and soil present 
off-white (seen in Fig. 3c). To get better color night vision effects, 
here we will present a new simple and effective color fusion meth¬ 
od in the Lab color space, which can enhance image quality and im¬ 
prove the target detection ability when bringing natural colors. The 
system is shown in Fig. 1. Based on the existed works, visible and 
IR images are preprocessed using contrast enhancement and med¬ 
ian filter. Consequently, the two enhanced images are fused into 
the three components of an Lab image by means of a simple linear 
fusion strategy. Different from the global statistic method, the 
transferring equation in the b channel is amended by a scaling fac¬ 
tor, which changes according to the distance between the current 
luminance value and the mean value. In this way the final render¬ 
ing image has overall clear and natural day-time color appearance 
when appropriate reference images are selected and the targets are 
easy to be recognized. This may help observers to improve situa¬ 
tional awareness and reduce detection and recognition time. 

2. Linear image fusion method 

Our fusion approach includes two steps: (1) pre-processing and 
(2) linear fusion. 

2.1. Pre-processing 

Visible images captured by charge coupled device (CCD) ther¬ 
mal cameras during night have dark background and low contrast 



Fig. 1 . A typical color image fusion system. 


due to a poorly illuminated environment. Generally most infrared 
(IR) images brought by thermal radiation have weak details and 
low contrast due to adverse weather and small temperature differ¬ 
ence between target objects and their background. To improve the 
accuracy of scene recognition, a pre-processing technique is 
needed before linear fusion. 

Histogram equalization (HE) is a simple and effective method of 
improving the quality of low contrast images that is done by 
stretching the dynamic range of an image’s gray level. HE methods 
can be divided into two classes: global and local methods. Global 
HE methods rescale pixel values in an input image (with the size 
ofMxN and L gray levels) to cover all of the available gray level 
range by a cumulative distribution function (CDF) t: 

k n 

t = Y-, n = MxN,k = 0,L,...,L-L (1) 

u n 

where ri; denotes the number of the pixels with gray level i. 

Global HE is effective in enhancing the low contrast image only 
if: (a) it contains single object or (b) no apparent contrast change 
between object and background. While the above conditions can¬ 
not be always met, the global methods have both over-enhance¬ 
ment and under-enhancement. Images with extra-high contrast 
may easily make observer tired and anxious, which are not suitable 
for long time observation. To overcome these drawbacks, we intro¬ 
duce local histogram equalization (LHE) method into enhancing 
visible and IR images as following steps: 

Step 1: segmenting the input image into several non-overlap¬ 
ping regions with the same size. 

Step 2: operating on these regions with CDF, rather than the 
entire image. 

Step 3: the neighboring regions are combined using bilinear 
interpolation to eliminate artificially induced boundaries. The 
new value at position (ij) is: 

f'(ij) = (1 - u)(l - + (1 - u)vf(ij +1) + u(l - v)/(f 

+ l,j) + uv/(i + lj + l) ( 2 ) 

where + 1),/(/ + 1 ,j) and f{i + l,j + 1) are values before 

interpolation in the enhancement image, u , v are constant, 
u, ve [0,1). 

Step 4: To avoid amplifying noise especially in homogeneous 
areas in the image, median filtering (MF) is used to smooth 
the enhancement image which can be described in short: 

g(i,j ) = median{f'(i - kj - /), (k, /) e A} (3) 

where g ( ij ) is the median value in the 5x5 neighborhood A 
around the corresponding pixel p (i,j) in the enhancement result 
f(i - k-l) is the gray value of pixels in window A. 

2.2. Linear image fusion method 

Normally, the false color fused image can be brought by a sim¬ 
ple linear fusion scheme mapping the IR and visible image to R, G 
and B channels. In this paper, we directly obtain the false color im¬ 
age in Lab color space in order to speed up this algorithm. Let I vis 
(visible image) and I iR (IR image) be two input night-vision images 
with the same spatial resolution and dynamic range. 

The false color fused image is given by: 

L = m,.I vis + m 2 .I ir 

a = Ivis ( 4 ) 

b = /ir — /vis 
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where L represents the lightness value, a and b are chrominance 
green-magenta and blue-yellow values separately, mi and m 2 are 
constant coefficient (mi + m 2 = 1), which make the IR and visible 
images be mapped to the achromatic channel. In this way, we can 
get a legible back ground because of much details in the visible im¬ 
age is preserved, and in the mean while the targets in the IR image 
are extruded. In addition, the differencing operation between the 
visible image and the IR image provides inter-sensor contrast. The 
resulting image agrees with our natural associations of warm (yel¬ 
low) and cool (blue). To further enhance the appearance of the fused 
results, the L, a and b channels can be input to the following remap¬ 
ping stage in which, they can be remapped to alternative “more nat¬ 
ural” display. 

3. Nonlinear color transfer 

The false color fused images of ten have unnatural color appear¬ 
ance and lack of color constancy. In this section, a simple technique 
is described in order to pop out targets and make the backgrounds 
full of natural colors. Different from literatures [2,7-13], the color 
transfer is directly applied to the device independent Lab (L*a*b* } 
CIE1976) color space. And to extrude cold and hot targets, b chan¬ 
nel will be enhanced by a scaling factor. 

In a color transfer technique, the fused color image should be 
transformed into a color space with reduced correlation compared 
to the RGB space. Lab model is a perceptually uniform color space 
and there is little correlation between the axes; so different oper¬ 
ations to different color channels can be applied with confidence 
that undesirable cross-channel artifacts will not occur. In addition, 
Lab is the most complete color model used conventionally to de¬ 
scribe all the colors visible to the human eye. There is no losing 
when an image is processed in Lab space. 

The first step of the color transfer technique is to transform the 
RGB tristimulus values of the day-light reference image to device 
independent XYZ ones as: 


X" 


"0.5141 0.3239 0.1604" 


"R" 

Y 

= 

0.2651 0.6702 0.0641 


G 

Z 


0.0241 0.1228 0.8444 


B 


And then to the Lab color space according to: 

( L = 116*/(Y/Y n ) - 16 

l a = 500*[f(X/X„)-/(Y/Y„)] (6) 

1 b = 200T(Y/Y„) -/(Z/Z„)] 

where the opposite colors are on the opposite sides of a line passing 
through the perfect white tristimulus values X n , Y n ,Z n . 


In the global color transfer, the three channels are processed 
with the same linear mapping as following: 


fuse 

< a' = 5N a-n% se ) 

fuse 

b ' = <( b - fi b u j 

>. fuse 


( 7 ) 


From Eq. (7), we can see that the values of each pixel in the fused 
image are globally modified by the same scaling factor cr re/ /cr /use . 
Hence, the intensity of the hot and cold targets in IR are weakened, 
which brings ambiguous targets and will reduce the recognition 
efficiency. To enhance the contrast between the targets and back¬ 
grounds, we propose a nonlinear color transfer scheme which pro¬ 
cesses each pixel at the basis of luminance distance value in the 
false color fused image. Because b represents the blue-yellow chan¬ 
nel, the larger its value, greater yellow color we can perceive; And 
the smaller its value, the greater blue color we can perceive. So to 
pop out hot and cold targets, only the b component in Eq. (7) is 
needed to be corrected. We modify the color transfer equation with 
a scaling factor w: 

b’ = w.^Li> (8) 

Vfncp 


And the scaling factor w(i, j) for each pixel (i, j) in the fused image is 
defined as: 


W(i,j) = dist(i,j)/ < dist > (9) 

M N 

< dist >=J2I2 d , ' sf ( y ) /mxn ( io ) 

M i= i 

dist(i,j) = \\L(i,j)-<L>)f (11) 

where dist(i,j ) denotes the luminance divergence of the pixel (i, j) 
from the global mean intensity of L channel in the fused image. 
L(i, j) and M x N are luminance value of the pixel (i, j)and the size 
of the fused image separately, p is a constant that can be modified 
according to the actual requirement for the enhancement, When p 
is set as 1, it become Yin’s method, so Yin’s method is only special 
case of our method. The effectiveness of color contrast enhance¬ 
ment become better along with the increasing of the parameter p 
in a certain range.In this way, the intensity of each pixel in the IR 
image can be expressed in the colorized result. And meanwhile 
the global transfer method brings the natural colors. Finally, we 
convert the result back to RGB. 



Fig. 2. Three sets of night-time images: (a) IR images and (b) visible images. 
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4. Colorizing results 

To validate the effectiveness of our method, three image sets are 
fused. For comparison, we also obtain the color fused images by 
Toet’s method without enhancement [7] and by Yin’s enhancement 
method [13]. The three image sets are shown in Fig. 2. Fig. 2a is IR 
images and Fig. 2b is corresponding visible images. 

Fig. 3 is the comparing experiment among our method, Toet’s 
and Yin’s methods. Fig. 3b-d are the colorizing results brought 


by Toet’s global statistic method, Yin’s method in YUV space with¬ 
out pre-processing and our method separately. It can be seen that 
the man takes on almost the same colors as the background and is 
very difficult to be detected without contrast enhancement and 
pre-processing in Fig. 3 b. But the man can be popped out using 
Yin’s and our methods. The man appears intense red in Fig. 3 c 
and intense yellow in Fig. 3d. When comparing with Yin’s method, 
tests show that both hot and cold targets are more outstanding in 
our rendering result. The man become bigger and brighter in 



(c) Yin’s method 


(d) our method 


Fig. 3. Fusion experiment of image pair NV1816. 



(c) Yin’s method (d) our method 


Fig. 4. Fusion experiment of image pair NV4917. 
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Fig. 3d. The cold targets such as the roof present light cyan in 
Fig. 3c, yet they obviously present deep blue in Fig. 3d. In addition, 
the details are clearer and the backgrounds can present more nat¬ 
ural colors in our result. The trees are green and the soil takes on 
suntan in Fig. 3d. While there are some red colors in the bottom 
of the Fig. 3c and the soil presents bright white using Yin’s method. 
The reason is that Lab have much broader color range than other 
color spaces. There is no losing when an image is processed in 
Lab space. And local histogram equalization and MF can eliminate 
noises and improve the image contrast. 

The same conclusions can be deduced from experiments using 
NV4917 pairs and NV7404 pairs (seen in Figs. 4 and 5). Without 


pre-processing and channel’s enhancement the man is hard to dis¬ 
tinguish from the background (e.g. Figs. 4b and 5b). Yin’s and our 
methods provide better results (seen in Figs. 4 and 5c and d). The 
man appears red and intense orange-yellow, thus can be easily 
distinguished from the backgrounds. 


5. Discussions 

Human visual inspection will be a good choice to evaluate col¬ 
orized images. However, the reproduction of subjective tests is of¬ 
ten time-consuming and difficult, while the exact same conditions 



(a) reference image 


(b) Toet’s method 



(c) Yin’s method (d) our method 

Fig. 5. Fusion experiment of image pair NV7404. 



Fig. 6. Detection results of different methods. 
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for the test cannot be guaranteed. Unfortunately, there is no effec¬ 
tive and well-accepted objective metric for color fusion schemes. 
However, as we aim at improving the identification ability of the 
targets by pre-processing and enhancing methods, we can roughly 
evaluate different color fusion methods through their detect abili¬ 
ties of the target [13]. Before giving the evaluation metrics, one 
precondition is to know the ideal target detection result, which is 
obtained manually from IR images in this paper. For color fusion 
images, we first produce different color fields by K-means cluster¬ 
ing, then adopt threshold segmentation to extract targets. In the 
case of Figs. 3-5, man is the observed hot target. The images in 
Figs. 6 are the segmented results of the above datas respectively. 
The first columns are the ideal targets. The detection results of 
Toet’s, Yin’s and our methods are shown from the second to the 
fourth columns. 

Then we evaluate different methods by calculating target’s rec¬ 
ognition area, detection rate, target-background discrimination. 
The recognition area s can be defined as the total number of pixels 
over the target field. The detection rate rate is described as: 


correct 

- num idea i 


( 12 ) 


where num correct denotes the sum of pixels detected correctly in the 
fused image, that is to say the corresponding pixels in the detected 
target to those in the ideal target. num ide a\ is the total number of 
pixels in the ideal target. 

The parameters 5 and rate only evaluate the targets detection 
ability from target itself, while they cannot embody the difference 
between the target and the background. When the target’s colors 
are very close to those of its surrounding background, it is difficult 
to distinguish although the detection rate may be large. So we also 
roughly evaluate different methods through their target-back¬ 
ground discrimination described by receiver-operator-characteris¬ 
tic (ROC) curves and color distance between the target and the 
background. 

A ROC curve is a plot of false alarm rate versus detection rate, 
calculated by determining what fraction of pixels are above and 
below varying threshold levels in HSV color space [14]. To obtain 
ROC curves, we first transform the color fused results from RGB 
color space to HSV space. Then target detection is performed 
through the hue and saturation components: 


tar(x,y) 


1 |H(x,y) -H\<H t and |S(x,y) - S\ < S T 

0 otherwise 


(13) 


where tar(x,y) is the target detection result; H(x,y) and S(x, y) are 
the hue and saturation components of the color fused image at 
the coordinate (x, y), respectively; H and S are the mean values of 
the hue and saturation of the target brought by K-means clustering, 
respectively; H T and S T are some given hue and saturation 
thresholds. 

Then the false alarm rate rate fa i se can be defined as: 


rate f aise 


numfaise 
all - numi dea i 


(14) 


Where numfai se denotes the sum of the pixels belonging to the back¬ 
ground and they are looked as the target by mistake using Eq. (13). 
all is the total number of the pixels in the color fused image. The 


Table 1 

Parameters setting for three sets. 



mi 

m 2 

Clustering number 

P 

Fig. 3 

0.5 

0.5 

7 

3 

Fig. 4 

0.35 

0.65 

7 

5 

Fig. 5 

0.7 

0.3 

7 

6 


ROC curves can be obtained with varying false alarm rates and tar¬ 
get detection rates brought by different hue and saturation thresh¬ 
olds for the target detection. 


Table 2 

Comparisons of recognition area and detection rate for different methods. 



S ideal 

Sglo 

5 Yin 

Sour 

rate g i 0 (%) 

rate (%) 

rate our (%) 

Fig. 3 

242 

183 

198 

237 

75.21 

81.82 

97.11 

Fig. 4 

113 

100 

85 

101 

84.96 

75.22 

89.38 

Fig. 5 

180 

171 

138 

135 

89.44 

76.67 

75.00 



Detection rate 



Detection rate 


0.25 



0 0.2 0.4 0.6 0.8 1 

Detection rate 


Fig. 7. ROC curves for the target-background discrimination with different image 
fusion methods: (a) ROC curves for the color fused results of image pair NV 1816; 
(b) ROC curves for the color fused results of image pair NV4917; (c) ROC curves for 
the color fused results of image pair NV7404. 
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Table 3 

Comparisons of region size color distance for different methods. 



Region size 

Dgio 

Dyin 

Dour 

Fig. 3 

90 x 110 

31.32 

67.68 

81.78 

Fig. 4 

90 x 110 

22.24 

68.86 

75.48 

Fig. 5 

90 x 110 

21.57 

63.43 

80.62 


Table 4 

Comparisons of full size color distance for different methods. 



Full size 

Dgio 

Dyin 

Dour 

Fig. 3 

277 x 209 

31.65 

69.82 

78.45 

Fig. 4 

208 x 105 

20.79 

69.40 

73.90 

Fig. 5 

280 x 209 

34.72 

74.06 

77.28 


The hit rate of color vision in detecting a target relies mainly on 
its ability to characterize color [15]. When the target is far from the 
background in color, it improves the object-background discrimi¬ 
nation. So we measure the color distance between the target region 
and its surrounding region. The distance D is defined as the Euclid¬ 
ean distance in Lab color space: 

D = y/uL,) - (4)) 2 + «a t ) - (a*)) 2 + «b t ) - (bf,)) 2 (15) 

where t, b stand for the target and the background regions, respec¬ 
tively. () is the mean value. 

Table 1 is the parameters setting for the three sets. They are the 
empirical values brought by several experiments. Table 2 shows 
the comparisons of recognition area and detection rate for different 
methods. The first column is the ideal target’s area. S gi0t S Y m and S our 
denote the target’s area of Toet’s colorizing method, Yin’s and our 
methods separately. And the next three columns give the detection 
rate of the three methods. Yin’s method enlarges the recognition 
area and improves the detection rate comparing with Toet’s global 
method, while the two parameters provided by our method are 
much closer to the ideal value, resulting in a performance improve¬ 
ment. We also see that the recognition area and detection rate are 
not very ideal from the third set (Fig. 5), because the parameter p 
has a larger value than that in Figs. 3 and 4. The larger is the 
parameter p, the more obvious the targets are. But the recognition 
area will be smaller. 

Fig. 7 gives the ROC curves for different methods. The varying 
hue and saturation thresholds are increased by step-length 0.005. 
It can be easily seen that the proposed method provides an 
improvement for target-background discrimination, as for the 
same false alarm rate, it obtains a higher detection rate. 

Tables 3 and 4 shows the comparisons of color distances in Lab 
space. Table 3 is the local comparison and the region size is 
90 x 110. Table 4 gives the global comparison for different meth¬ 
ods. Yin’s method enlarges the distance between the target and 
the background drastically due to the intense red colors of the tar¬ 
gets. While our method shows better performance from local and 
global comparisons. The color distance between the targets with 
the intense yellow colors brought by our method and the back¬ 
grounds has three to four times than that of Toet’s global method 


without channel’s contrast enhancement. It also can be seen that 
the color distance brought by our method also gets improvement 
comparing to that of Yin’s. Besides, our method raises the target’s 
recognition area and detection rate seen from Table 2. 

6. Conclusions 

In this paper, we discuss a simple and effective color contrast 
enhancement method for color night vision. Before fusion, local 
histogram equalization and median filter are introduced to im¬ 
prove the visible and IR images’ quality. Then a simple linear fusion 
method is proposed. In order to get natural colors and improve the 
target detection ability, an amended global color transfer method 
is applied. Under the control of the scaling factors, the targets 
can be popped out and meanwhile the backgrounds are rendered 
with natural colors. Visual experiments and object metrics all show 
that our method has a better performance. 
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